Online compensation of thermal distortions in a stereo depth camera

ABSTRACT

An example apparatus includes: a camera to record an image; memory to store instructions; and a processor in circuit with the memory, the processor to execute the instructions to: determine a depth based on: (a) the image and (b) a calibration parameter of the camera; and adjust the calibration parameter based on a temperature of the camera and the depth.

CROSS-REFERENCE TO RELATED APPLICATIONS

This patent arises from a continuation of U.S. patent application Ser.No. 16/450,524, filed Jun. 24, 2019, entitled “ONLINE COMPENSATION OFTHERMAL DISTORTIONS IN A STEREO DEPTH CAMERA,” which is a continuationof U.S. patent application Ser. No. 15/805,598, filed Nov. 7, 2017,entitled “ONLINE COMPENSATION OF THERMAL DISTORTIONS IN A STEREO DEPTHCAMERA,” now U.S. Pat. No. 10,390,002, which is a continuation of U.S.patent application Ser. No. 14/865,196, filed Sep. 25, 2015, entitled“ONLINE COMPENSATION OF THERMAL DISTORTIONS IN A STEREO DEPTH CAMERA,”now U.S. Pat. No. 9,813,692. U.S. patent application Ser. No.16/450,524, U.S. patent application Ser. No. 15/805,598 and U.S. patentapplication Ser. No. 14/865,196 are hereby incorporated herein byreference in their entireties.

FIELD OF THE INVENTION

Embodiments of the present invention relate to the field of camerasystems that perform three-dimensional (3-D) measurements by use ofactive or passive triangulation capturing one or more images with orwithout the assistance of one or more patterns projected by a projector;more particularly, embodiments of the present invention relate toperforming temperature compensation of such camera systems.

BACKGROUND OF THE INVENTION

Stereo depth cameras are well-known and are often used to measure adistance from an object. As part of the process, images of scenes arecaptured and measurements are taken to determine depth information. Onemethod of determining the depth of objects in a scene from a capturedimage is made using triangulation.

In order to perform triangulation accurately, some calibration of thestereo depth camera is typically performed at assembly time. Calibrationsets some intrinsic and extrinsic parameters of the stereo camera.Subsequent calibrations may be needed due to the system's deviation frominitial calibration due to various external factors, such astemperature, humidity or barometric pressure changes, or displacement ofsensing and/or projecting devices included in the system.

Stereo depth cameras are very sensitive to tiny changes in the opticelements intrinsic (e.g., Field of View or focal length) and extrinsicparameters. As a result of these changes, key depth imagecharacteristics are biased such as absolute depth value, worldcoordinate y-axis yaw, scaling effects and more. Temperature variationsmay cause changes in the parameters, thereby causing errors in thedetermined depth generated by the triangulation process. Currentsolutions to these problems caused by temperature either ignore thissource of distortion, or apply active thermal control to maintain thetemperature at which the system was initially calibrated. The formerapproach is very weak for a real product since it implies a narrowworking temperature range and the system being at the “working point”where the temperature is stable. The active thermal control allowscorrect operation in a wider range of temperatures, yet it consumespower, area and cost, which are all undesirable.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will be understood more fully from the detaileddescription given below and from the accompanying drawings of variousembodiments of the invention, which, however, should not be taken tolimit the invention to the specific embodiments, but are for explanationand understanding only.

FIG. 1 illustrates one embodiment of an active coded light triangulationsystem.

FIG. 2 is a block diagram depicting one embodiment of an onlinecompensation scheme for a stereo depth camera.

FIG. 3 is a flow diagram of one embodiment of a process for performingonline temperature compensation for thermal distortions in a stereodepth camera.

FIG. 4 illustrates an example of coefficient precomputation parameters.

FIG. 5 illustrates an example of range and texture reconstruction andcoefficient precomputation.

FIG. 6 is a block diagram of one embodiment of a system.

FIG. 7 illustrates an embodiment of a computing environment 700 capableof supporting the operations.

DETAILED DESCRIPTION OF THE PRESENT INVENTION

In the following description, numerous details are set forth to providea more thorough explanation of the present invention. It will beapparent, however, to one skilled in the art, that the present inventionmay be practiced without these specific details. In other instances,well-known structures and devices are shown in block diagram form,rather than in detail, in order to avoid obscuring the presentinvention.

The description may use the phrases “in an embodiment,” or “inembodiments,” which may each refer to one or more of the same ordifferent embodiments. Furthermore, the terms “comprising,” “including,”“having,” and the like, as used with respect to embodiments of thepresent disclosure, are synonymous.

The term “coupled with,” along with its derivatives, may be used herein.“Coupled” may mean one or more of the following. “Coupled” may mean thattwo or more elements are in direct physical, electrical, or opticalcontact. However, “coupled” may also mean that two or more elementsindirectly contact each other, but yet still cooperate or interact witheach other, and may mean that one or more other elements are coupled orconnected between the elements that are said to be coupled with eachother. The term “directly coupled” may mean that two or more elementsare in direct contact.

FIG. 1 illustrates one embodiment of an active coded light triangulationsystem. The system includes coded light range cameras operating byprojecting a sequence of one-dimensional binary (“black” and “white”)patterns onto a scene, such that the produced binary code encodes theangle of the projection plane. Depth is then reconstructed bytriangulating the intersection of a ray emanating from the camera withthe plane emanating from the projector.

Referring to FIG. 1, capture device 100 may include a 3D scanner, a 3Dcamera or any other device configured for a 3D object acquisition. Insome embodiments, as illustrated, capture device 100 includes an imagecapturing device 102 (e.g., a digital camera) and a projector unit 104,such as a laser projector or laser scanner, having a number ofcomponents. In some embodiments, digital camera 102 may comprise aninfrared (IR) camera, and the projector unit 104 may comprise an IRprojector.

Projector unit 104 is configured to project a light pattern as describedabove and may comprise a one-dimensional coded projector. In oneembodiment, the light patterns comprise one-dimensional coded lightpatterns, e.g., the patterns that may be described by one-dimensional orlinear codes. The light patterns formed by the laser planes on a surfaceof the object may be received by image capturing device 102 and sensed(e.g., read) by a sensor of image capturing device 102. Based on thereadings of the multiple scans of the light patterns accumulated duringa sensing cycle of the sensor, capture device 100 may be configured toreconstruct the shape of the object.

In some embodiments, capture device 100 may further include anotherimage capturing device, such as digital camera 103. In some embodiments,digital camera 103 may have a resolution that is different than that ofdigital camera 103. For example, digital camera 102 may be amulti-chromatic camera, such as red, green, and blue (RGB) cameraconfigured to capture texture images of an object.

Capture device 100 may further include a processor 106 that may be inoperative communication with the image camera component 101 over a busor interconnect 107. Processor 106 may include a standardized processor,a specialized processor, a microprocessor, or the like that may executeinstructions that may include instructions for generating depthinformation, generating a depth image, determining whether a suitabletarget may be included in the depth image, or performing otheroperations described herein.

Processor 106 may be configured to reconstruct the object based on theimages captured by digital camera 102, for example, using geometrytechniques or other techniques used for 3D image reconstruction.Processor 106 may be further configured to dynamically calibrate capturedevice 100 to correct distortions in the reconstructed image of theobject that may be caused, for example, by various external factors(e.g., temperature).

Capture device 100 may further include a memory 105 that may store theinstructions that may be executed by processor 106, images or frames ofimages captured by the cameras, user profiles or any other suitableinformation, images, or the like. According to one example, memory 105may include random access memory (RAM), read only memory (ROM), cache,Flash memory, a hard disk, or any other suitable storage component. Asshown in FIG. 1, memory component 105 may be a separate component incommunication with the cameras 101 and processor 106. Alternatively,memory 105 may be integrated into processor 106 and/or the image capturecameras 101. In one embodiment, some or all of the components 102-106are located in a single housing.

Processor 105, memory 104, other components (not shown), image capturingdevice 102, and projector unit 104 may be coupled with one or moreinterfaces (not shown) configured to facilitate information exchangeamong the above-mentioned components. Communications interface(s) (notshown) may provide an interface for device 100 to communicate over oneor more wired or wireless network(s) and/or with any other suitabledevice. In various embodiments, capture device 100 may be included to orassociated with, but is not limited to, a server, a workstation, adesktop computing device, or a mobile computing device (e.g., a laptopcomputing device, a handheld computing device, a handset, a tablet, asmartphone, a netbook, ultrabook, etc.).

In one embodiment, capture device 100 is integrated into a computersystem (e.g., laptop, personal computer (PC), etc.). However, capturedevice 100 can be alternatively configured as a standalone device thatis couplable to such a computer system using conventional technologiesincluding both wired and wireless connections.

In various embodiments, capture device 100 may have more or lesscomponents, and/or different architectures. For example, in someembodiments, capture device 100 may include one or more of a camera, akeyboard, display such as a liquid crystal display (LCD) screen(including touch screen displays), a touch screen controller,non-volatile memory port, antenna or multiple antennas, graphics chip,ASIC, speaker(s), a battery, an audio codec, a video codec, a poweramplifier, a global positioning system (GPS) device, a compass, anaccelerometer, a gyroscope, and the like. In various embodiments,capture device 100 may have more or less components, and/or differentarchitectures. In various embodiments, techniques and configurationsdescribed herein may be used in a variety of systems that benefit fromthe principles described herein.

Capture device 100 may be used for a variety of purposes, including, butnot limited to, being part of a target recognition, analysis, andtracking system to recognize human and non-human targets in a capturearea of the physical space without the use of special sensing devicesattached to the subjects, uniquely identify them, and track them inthree-dimensional space. Capture device 100 may be configured to capturevideo with depth information including a depth image that may includedepth values via any suitable technique including, for example,triangulation, time-of-flight, structured light, stereo image, or thelike.

Capture device 100 may be configured to operate as a depth camera thatmay capture a depth image of a scene. The depth image may include atwo-dimensional (2D) pixel area of the captured scene where each pixelin the 2D pixel area may represent a depth value such as a distance in,for example, centimeters, millimeters, or the like of an object in thecaptured scene from the camera. In this example, capture device 100includes an IR light projector 404, an IR camera 102, and a visiblelight RGB camera 103 that are configured in an array.

Various techniques may be utilized to capture depth video frames. Forexample, capture device 100 may use structured light to capture depthinformation. In such an analysis, patterned light (i.e., light displayedas a known pattern such as a grid pattern or a stripe pattern) may beprojected onto the capture area via, for example, IR light projector104. Upon striking the surface of one or more targets or objects in thecapture area, the pattern may become deformed in response. Such adeformation of the pattern may be captured by, for example, the IRcamera 102 and/or the RGB camera 103 and may then be analyzed todetermine a physical distance from capture device 100 to a particularlocation on the targets or objects.

Capture device 100 may utilize two or more physically separated camerasthat may view a capture area from different angles, to obtain visualstereo data that may be resolved to generate depth information. Othertypes of depth image arrangements using single or multiple cameras canalso be used to create a depth image.

Capture device 100 may provide the depth information and images capturedby, for example, IR camera 102 and/or the RGB camera 103, including askeletal model and/or facial tracking model that may be generated bycapture device 100, where the skeletal and/or facial tracking models,depth information, and captured images are used to, for example, createa virtual screen, adapt the user interface, and control an application.

In summary, capture device 100 may comprise a projector unit 104, adigital camera (e.g., IR camera) 102, another digital camera (e.g.,multi-chromatic camera) 103, and a processor (controller) configured tooperate capture device 100 according to the embodiments describedherein. However, the above assembly configuration is described forillustration purposes only, and is should not be limiting to the presentdisclosure. Various configurations of an assembly for a 3D objectacquisition may be used to implement the embodiments described herein.For example, an assembly for a 3D object acquisition configured toenable the reconstructed object distortion corrections may include threedigital cameras, two of which may be used to reconstruct a 3D image ofan object, and the third camera (e.g. with a resolution that isdifferent than those of the two cameras) may be used to capture imagesof the object in order to identify image distortions in thereconstructed object and to compensate for identified distortions.

Real-Time Thermal Distortion Compensation

Techniques described herein are directed to the problem of real-timecorrection of temperature-related drifts in the intrinsic and extrinsicgeometric parameters of a stereo depth camera. In one embodiment, thetechnique includes a thermal process that includes measuring the currenttemperature of the capture device (e.g., capture device 100 of FIG. 1),or part thereof (e.g., IR camera, or near such a device), at one orseveral points, and applying a correction model on the fly tocalibration parameters used in the process for generating depth or othervalues. In one embodiment, the process is the triangulation process. Theresult is a depth image with considerably diminished thermaldistortions.

These techniques have a number of benefits such as passive sensing ofthe temperature and open loop fix on the result depth parameters. Thetechniques are both reasonably accurate and cost effective and can copewith a large range of ambient and self-heat temperatures.

More specifically, as discussed above, a coded light camera comprisingan IR projector 104 projects one-dimensional code patterns onto thescene, and an IR camera 102 captures the patterns. Decoding of thecaptured patterns at every pixel location x_(c) in the camera produces acode encoding the location x_(p) of the projected plane. Intriangulation, the plane is intersected with the ray emanating from thecamera focal point through x_(c), yielding the distance to the objectz(x_(c)). In one embodiment, triangulation relies on two types ofparameters: the intrinsic parameters of camera 102 and the projector 104characterizing their individual optical systems (such as focal distance,principal point of the projection, and a distortion model accounting fordeviations from the perfect pin-hole model), and the extrinsicparameters specifying the relative location and orientation of projector104 with respect to the camera (e.g., IR camera) 102.

For purposes herein, the intrinsic camera and projector parameters arecollectively noted as θ_(c) and θ_(p), respectively, while the extrinsicparameters are denoted as θ_(e). These three sets of parameters arefurther collectively denoted as θ. The triangulation process can bewritten symbolically asz=T(x _(c) ,x _(p);θ):

The parameters θ=(θ_(c);θ_(p);θ_(e)) are specific to a device and in oneembodiment are determined by use of a calibration procedure, which istypically a part of the unit assembly process, or a global parametricmodel for the system.

However, during the operation of capture device 100, mechanical changesdue to external or internal heating may change the parameters to adegree impacting adversely the quality of depth reconstruction. Forexample, uneven heating of the lens may cause its vertical shiftimpacting the focal length, or a rotation, impacting the principalpoint. Furthermore, the whole assembly may bend, impacting theconvergence angle between the camera (e.g., camera 102) and theprojector (e.g., projector 104).

Techniques described herein include a way to overcome this drawback ofcommon designs by creating a parametric model that receives, asinput(s), the reading of the unit temperature at one or more points incapture device 100, and produces as the output a correction of the unitintrinsic and extrinsic calibration parameters that are updated on thefly and used for triangulation. Denoting the temperature input as t,this can be written symbolically as:Δθ=C(t;a);where a are the model parameters and Δθ is the correction.

FIG. 2 is a block diagram depicting one embodiment of an onlinecompensation scheme for a stereo depth camera. The stereo depth cameracomprise a projector configured to project a sequence of light patternson an object, a first camera configured to capture a sequence of imagesof the object illuminated with the projected light patterns, and one ormore temperature sensors to measure the temperature at one or morelocations in the stereo camera. In one embodiment, a temperature sensoris attached to a printed circuit board (PCB) and located near the IRcamera on the PCB to measure temperature at that location. In anotherembodiment, an additional temperature sensor is located on the PCB inthe middle between the IR camera and the IR projector. In yet anotherembodiment, an additional temperature sensor is included near the IRprojector on the PCB.

Referring to FIG. 2, an image processing pipeline 301 receives asequence of images and determines depth information in response toparameters of the projector (e.g., projector 104) and camera (e.g.,camera 102). Note that in one embodiment, image processing pipeline 201of FIG. 2 is part of the capture device (e.g., the capture device ofFIG. 1). In another embodiment, the image processing pipeline of FIG. 2is not part of the capture device and performs triangulation in aseparate device.

Image processing pipeline 201 includes a triangulation processing unit210. Triangulation processing unit 210 receives x_(p) and x_(c), whichare well-known in the triangulation process and calibration parametersand, in response thereto, generates depth values, z, by means oftriangulation. The value of x_(p), which corresponds to a pixel, isreceived as an input at the triangulation-based reconstruction module.This value includes a decoded value of a pixel location of the pixel ona projector plane, as determined from a binary code, which, in turn, isdetermined based on the image captured by the IR camera. Thetriangulation-based reconstruction process is configured to take intoaccount the distortion due to temperature.

In one embodiment, triangulation processing unit 210 receives thecalibration parameters from a thermal correction unit 220. In oneembodiment, the calibration parameters characterize an optical systemformed by the projector and the first camera and comprise intrinsicparameters of the first camera (e.g., IR camera) and the projector(e.g., IR projector) and extrinsic parameters of first camera (e.g., IRcamera). In one embodiment, the intrinsic parameters comprise one ormore of a focal distance of an IR camera, a principal point ofprojection of an IR projector, and a distortion model and wherein theextrinsic parameters comprise a relative location and orientation of theIR projector with respect to the IR camera.

Thermal correction unit 220 is responsive to temperature information 231from the one or more temperature sensors 230 on the capture device(e.g., the capture device of FIG. 1). In response to the temperatureinformation, thermal correction unit 220 adjusts one or more of thecalibration parameters of the projector and the first camera that areprovided to and used by triangulation processing unit 210 in generatingthe depth values.

In one embodiment, the calibration parameters are initialized by one ormore calibration procedures and thermal correction unit 220 is operableto adjust one or more of these calibration parameters based on outputsfrom a thermal correction model 242 that is responsive to temperatureinformation 231 from one or more temperature sensors 230.

In one embodiment, thermal correction unit 220 receives an initial setof calibration parameters θ (250) and temperature information 231indicative of the temperature of at least a portion of the capturedevice (e.g., the temperature near the IR camera, the temperature at alocation on the printed circuit board of the capture device between theIR camera and the IR projector, the temperature near the IR projector,etc.). In one embodiment, the initial set of calibration parameters θ(250) are those calibration parameters determined during assembly ormanufacturing of the capture device (e.g., prior to deployment). Thisset of calibration parameters are set under specific temperatureconditions that existed during the initial calibration process (i.e.,the temperature at which the capture device is calibrated).

Thermal correction model 242 receives temperature information 231 andgenerates values (Δθ 251) for modifying calibration parameters in theinitial set of calibration parameters θ 250. Computation unit 241 (e.g.,adder) modifies the initial set of calibration parameters θ 250 based onthe Δθ 251 output from thermal correction model 242 to produce a set ofcalibration parameters 252 that are input to triangulation unit 210.

In one embodiment, thermal correction model 242 is device-specific andcan be generated by testing a device under different temperatures andrecording changes for each individual calibration parameters (Δθ) foreach specific change in temperature. For example, a capture device couldbe attached to a thermo-electric controller and be subjected to avariety of different temperatures to see their effect on the camerasand/or projectors in the device, with focus on the parameters that arebased on the depth value generation process (e.g., triangulation). Onceall the changes have been recorded, a function can be defined oridentified that models the behavior of the device under those differenttemperature conditions. In one embodiment, graphs are generated thatplot changes to each parameter versus the change in temperature (fromthe nominal temperature under which the capture device was originallycalibrated). For each plot, a function that best represents the pointsin the plot is chosen. For example, in one embodiment, a linear modelmay represent the behavior of a particular parameter under differenttemperatures. In such a case, a coefficient α associated with generatinga Δθ for that parameter may be set equal to the slope of the linerepresenting the linear model. This may be shown with the followingequation:Δθ₁(t)=α₁(t−t ₀)where a₁ is a coefficient representing the effect of temperature on theparameter θ₁ (e.g., the slope of the linear function modeling thebehavior of the parameter θ₁ for various temperatures) from thedifference in temperature between the current temperature t andtemperature t₀, which is the nominal temperature under which the initialparameter θ₁ use set and determined during initial calibration. Whilethis example uses a linear model, the generation of Δθ values forcompensation may be based on non-linear models, at least in part becausethese values are device and implementation dependent.

Note the formula above only mentions one coefficient; however, ingeneral, there will be a vector α containing a number of coefficients,one for each parameter that is to undergo compensation due totemperature.

In one embodiment, taking temperature measurements and the updating ofΔθ values occurs every frame according to the frame rate. Thus, changesidentified during one frame are used in processing the next fame.

FIG. 3 is a flow diagram of one embodiment of a process for performingonline temperature compensation for thermal distortions in a stereodepth camera. The process is performed by processing logic that maycomprise hardware (circuitry, dedicated logic, etc.), software (such asis run on a general purpose computer system or a dedicated machine),firmware, or a combination of these three.

Referring to FIG. 3, the process begins by projecting a sequence oflight patterns on an object using a projector (processing block 301). Inone embodiment, this is performed by projector 104 (e.g., an IRprojector) of FIG. 1.

A camera captures a sequence of images of the object illuminated withthe projected light patterns (processing block 302). In one embodiment,the camera is camera 102 (e.g., an IR camera) of FIG. 1.

From the captured sequence of images, processing logic determines depthinformation using calibration parameters of the projector and camera,including adjusting, based on temperature information from one or moretemperature sensors, one or more of the calibration parameters of theprojector and the first camera (processing block 303). In oneembodiment, the calibration parameters characterize an optical systemformed by the projector and the first camera and comprise intrinsicparameters of the first camera and the projector and extrinsicparameters of first camera. In one embodiment, the intrinsic parameterscomprise one or more of a focal distance of the first camera, aprincipal point of projection of the projector, and a distortion modeland wherein the extrinsic parameters comprise a relative location andorientation of the projector with respect to the first camera.

In on embodiment, determining depth information comprises performing atriangulation process using the calibration parameters. In oneembodiment, the calibration parameters are initialized by one or morecalibration procedures and adjusting calibration parameters of theprojector and the camera is based on outputs from a thermal correctionmodel that is responsive to the temperature information from the one ormore temperature sensors.

FIG. 3 described above refers only to a time-multiplexed structural(coded) light system. The technologies described herein are applicableto other systems and ways of operation (e.g., passive stereo). Forexample, such systems may include passive stereo: 1) passive stereosystems comprising two (or more) calibrated cameras; 2) assistedstereo—as above plus an uncalibrated projector projecting some type ofrandom noise onto the scene to improve correspondence computationbetween the cameras; and 3) coded light system comprising one (or more)calibrated camera and a calibrated projector projecting code. The lattercode can be spatially multiplexed (single pattern), temporallymultiplexed (multiple patterns), or both. The code can be onedimensional (encoding planes) or two-dimensional (encoding rays) or acombination of the two.

A Triangulation Process with an Exemplary Forward Model

Pattern Camera

Given a point x in world coordinates, it is projected onto the cameraimage plane by the following pin-hole model:x _(c) ′=K _(c)(I0)x  (1)where x_(c) is a 4×1 vector representing the world point location inhomogenous coordinates, K_(c) is the 3×3 upper-triangular intrinsiccamera matrix,

$\begin{matrix}{{K_{c} = \begin{pmatrix}f_{x} & \; & C_{x} \\\; & f_{y} & {Cy} \\\; & \; & 1\end{pmatrix}},} & (2)\end{matrix}$and x_(c) is the 3×1 vector of uncorrected homogenous cameracoordinates. The camera system of coordinates is assumed to coincidewith the world coordinate system, therefore trivial rotation andtranslation (extrinsic) matrices are imposed.Pattern Camera Lens Distortion

To account for the lens distortion, a parametric distortion model isapplied. The corrected camera coordinates vector x_(c) is related to itsuncorrected counterpart x_(c)′ through the following inverse model givenbyx _(c) ′=K _(c) D _(α)(K _(c) ⁻¹ x _(c))  (3)where D is a plane-to-plane map given byx′=(1+α₁ p ²+α₂ p ⁴+α₃ p ⁶)x+2α₄ ry+α ₅(p ²+2x ²)y′=(1+α₁ p ²+α₂ p ⁴+α₃ p ⁶)y+2α₅ ry+α ₄(p ²+2y ²)  (4)with p²=x′²+y′². The parameters α₁, α₂, α₃ govern the radial distortioncomponent, while α₄ and α₅ account for the tangential one. The vectorα=(α₁, . . . α₅)^(T) is referred to herein as to the camera inversedistortion parameters. Note that the difference between this inversemodel versus the standard forward model used in other systems. Thebenefits of the teachings described herein will be evident when thereconstruction process is disclosed.Projector

The projector is modeled as a one-dimensional pin-hole systemx _(p) =K _(p)(R _(p) t _(p))x=P _(t) x,  (5)where x_(p) is the 2×1 vector of homogenous coordinate on the projectorline, K_(p) is the 2×3 intrinsic matrix, and R_(p) and t_(p) are theextrinsic rotation and translation transformation Parameters,respectively, relating between the camera and the projector coordinatesystems. In one embodiment, the intrinsic and the extrinsic projectorparameters are combined into a single 2×4 projection matrix P_(p).Texture Camera

The texture camera is modeled as a two-dimensional pin-hole systemx _(t) ′=K _(t)(R _(t) t _(t))x=P _(t) x,  (6)where x_(t) is the 3×1 uncorrected vector of homogenous coordinate onthe camera plane, K_(t) is the 3×3 intrinsic matrix, and R_(t) and t_(t)are the extrinsic rotation and translation transformation parameters,respectively, relating between the pattern and the texture camerascoordinate systems. In one embodiment, the intrinsic and the extrinsictexture camera parameters are combined into a single 3×4 projectionmatrix P_(t).Texture Camera Lens Distortion.

To account for the lens distortion, a forward parametric distortionmodel is applied:x _(t) =K _(t) D _(β)(K _(t) ⁻¹ x _(t)′)  (7)where D is a plane-to-plane map given byx=(1+β₁ p ²+β₂ p ⁴+β₃ p ⁶)x′+2β₄ x′y′+β ₅(p ²+2x′ ²)y=(1+β₁ p ²+β₂ p ⁴+β₃ p ⁶)y′+2β₅ x′y′+β ₄(p ²+2x′ ²)  (8)with p²=x′²+y′². The parameters β₁, β₂, β₃ govern the radial distortioncomponent, while β₄ and β₅ account for the tangential one. The vectorβ=(β₁, . . . β₅)^(T) is referred to herein as the texture camera forwarddistortion parameters. The forward model is applied by transformingdirectly the texture image using D_(β). In one embodiment, the processis performed on the host system due to the requirement of random accessto the frame buffer.Depth Reconstruction

The purpose of depth reconstruction is to establish for each pixel inthe pattern camera image its distance from the camera center ofcoordinates. The term “depth” shall be understood as either the distancer or its projection z onto the optical axis of the pattern camera.

Given a pixel at position x_(c)=(x_(c), y_(c),1)^(T) in the camera imageplane and the corresponding code value x_(p)=(x_(p),1)^(T) of theprojector, by applying the inverse distortion model (7), a correctedpixel location x_(c)′=(x_(c)′,y_(c)′,1)^(T) is obtained. The pinhonecamera projection (1) introduces ambiguity by projecting all pointsalong the ray x=rd, r>0, to x_(c)′.

In order to characterize the ray d=(d_(x); d_(y); d_(z))^(T), thefollowing is writtenβx _(c) =K _(c) d,  (9)where β≠0 is an arbitrary scaling factor. This yieldsd _(z) x _(c)′=(K _(c))₁ ^(T) d=f _(x) d _(x) +c _(x) d _(z)d _(z) y _(c)′=(K _(c))₂ ^(T) d=f _(y) d _(y) +c _(y) d _(z)  (10)where (K_(c))_(i) ^(T) denotes the i-th row vector of K_(c). Solving ford_(x); d_(y) and d_(z) gives

$\begin{matrix}{d = \begin{pmatrix}{f_{x}^{- 1}\left( {x_{c}^{\prime} - c_{x}} \right)} \\{f_{y}^{- 1}\left( {y_{c}^{\prime} - c_{y}} \right)} \\1\end{pmatrix}} & (11)\end{matrix}$Since the pixel grid is fixed at reconstruction, the described processis precomputed for every pixel in the image.

The projector forward model (6) introduces ambiguity by projecting anentire plane onto x_(p); however, the intersection of the latter planewith the ray x=rd yields a unique point in space. Substituting the unitdirection d=d/∥d∥ into the projector model gives

$\begin{matrix}{{{\beta x_{p}} = {P_{p}\begin{pmatrix}{rd} \\1\end{pmatrix}}},} & (12)\end{matrix}$which can be rewritten as

$\begin{matrix}{{\beta x_{p}} = {\frac{{rp_{1}^{T}d} + {1q_{1}}}{{rp_{2}^{T}d} + {1q_{2}}} = \frac{{rp_{1}^{T}d} + {1q_{1}{d}}}{{rp_{2}^{T}d} + {1q_{2}{d}}}}} & (13)\end{matrix}$where the projector matrix is decomposed into

$\begin{matrix}{P_{p} = {\begin{pmatrix}{p_{1}^{T}q_{1}} \\{p_{2}^{T}q_{2}}\end{pmatrix}.}} & (14)\end{matrix}$Solving for r, one has the depth

$\begin{matrix}{r = \frac{{T_{1}x_{p}} + T_{2}}{{T_{3}x_{p}} + T_{4}}} & (15)\end{matrix}$whereT ₁ =q ₂ ∥d∥T ₂ =−q ₁ ∥d∥T ₃ =p ₂ ^(T) dT ₄ =−p ₁ ^(T) d  (16)are scalar coefficients depending only on the pixel location x_(c) (andtherefore can be precomputed). For purposes herein T₁, . . . . T₄ arereferred to as to range triangulation coefficients. Note that using theunnormalized ray saves one division operation.

The depth r can be further projected onto d_(z) yielding

$\begin{matrix}{{z = \frac{{T_{1}x_{p}} + {T_{2}d_{z}}}{{T_{3}x_{p}} + {T_{4}{d}}}}{= \frac{{T_{1}^{\prime}x_{p}} + T_{2}^{\prime}}{{T_{3}x_{p}} + T_{4}}}} & (17)\end{matrix}$ $\begin{matrix}{{where}{T_{1}^{\prime} = {q_{2}d_{z}}}{T_{2}^{\prime} = {{- q_{1}}d_{z}}}} & (18)\end{matrix}$are the modified coefficients.Texture Map Reconstruction

Once the range has been established, the texture map coordinates arereconstructed using the forward model

$\begin{matrix}{x_{t} = {{P_{t}\begin{pmatrix}{rd} \\1\end{pmatrix}} = {P_{t}\begin{pmatrix}{rd} \\{d}\end{pmatrix}}}} & (19)\end{matrix}$from where

$\begin{matrix}{{x_{t} = \frac{{rp_{1}^{T}d} + {q_{1}{d}}}{{rp_{3}^{T}d} + {q_{3}{d}}}}{y_{t} = \frac{{{rp}_{2}^{T}d} + {q_{2}{d}}}{{{rp}_{3}^{T}d} + {q_{3}{d}}}}} & (20)\end{matrix}$where (p_(i) ^(T)q_(i)) is the i-th row of P_(t). This can be rewrittenas

$\begin{matrix}{{x_{t} = \frac{{\sigma_{1}r} + \sigma_{4}}{{\sigma_{3}r} + \sigma_{6}}}{{y_{t} = \frac{{\sigma_{2}r} + \sigma_{5}}{{\sigma_{3}r} + \sigma_{6}}},}} & (21)\end{matrix}$ $\begin{matrix}{{where}{\sigma_{1} = {p_{1}^{T}d}}{\sigma_{2} = {p_{2}^{T}d}}{\sigma_{3} = {p_{3}^{T}d}}{\sigma_{4} = {q_{1}{d}}}{\sigma_{5} = {q_{2}{d}}}{\sigma_{6} = {q_{3}{{d}.}}}} & (22)\end{matrix}$

In case the output depth is projected onto the camera optical axis,texture coordinates are obtained from r=z∥d∥/d_(z),

$\begin{matrix}{{x_{t} = \frac{{zp_{1}^{T}d{d}} + {q_{1}{d}d_{z}}}{{rp_{3}^{T}d} + {q_{3}{d}}}}\frac{{zp_{1}^{T}} + {q_{1}d_{z}}}{{zp_{3}^{T}d} + {q_{3}d_{z}}}} & (23)\end{matrix}$and, similarly,

$\begin{matrix}{y_{t} = \frac{{zp_{2}^{T}d} + {q_{2}d_{z}}}{{zp_{3}^{T}d} + {q_{3}d_{z}}}} & (24)\end{matrix}$This can be expressed as (21) with σ₄, σ₅ replaced by σ₆ replaced byσ₄ ′=q ₁ d _(z)σ₅ ′=q ₂ d _(z)σ₆ ′=q ₃ d _(z).  (25)Coefficient Scaling and Precomputation

In the following, a compact and efficient way to compute thecoefficients T_(i) and σ_(i) is disclosed. It assumes the pattern cameracoordinates to be unnormalized and translated from [−1, 1]² to[0,W−1]×[0,H−1]; the projector coordinate is assumed to be translatedfrom [1, 1] to [0; s₂−1] (s₂=2¹⁵−1 in one embodiment); the output rangeor depth is mapped from the interval [0; r_(max)] to [0, 2¹⁶−1] by thescaling factor s₁; and the texture map coordinates are computedtranslated from [−1, 1] to [0, s_(uv)] with s_(uv)=2¹²−1 in oneembodiment.

Inverse Camera Distortion Model.

Substituting the scales into the camera distortion model, the followingis obtained with appropriate scaling

$\begin{matrix}{{x = {{\frac{2}{Wf_{x}}x_{c}} - {\frac{H}{2}\frac{f_{y}}{f_{x}}\left( {c_{x} + 1} \right)}}}{y = {y_{c} - {\frac{H}{2}{\left( {c_{y} + 1} \right).}}}}} & (26)\end{matrix}$The distortion model becomesx′=(1+d ₁ p ² +d ₂ p ⁴ +d ₃ p ⁶)x+2d ₄ xy+d ₅(p ²+2x ²)y′=(1+d ₁ p ² +d ₂ p ⁴ +d ₃ p ⁶)y+2d ₅ xy+d ₄(p ²+2y ²)  (27)p²=x²+y² and d_(i) are the distortion parameters scaled according to

$\begin{matrix}{{{d_{1} = {\left( \frac{2}{Hf_{y}} \right)^{2}\alpha_{1}}}{d_{2} = {\left( \frac{2}{Hf_{y}} \right)^{4}\alpha_{2}}}{d_{3} = \alpha_{3}}{d_{4} = {\left( \frac{2}{Hf_{y}} \right)^{2}\alpha_{4}}}{d_{5} = {\left( \frac{2}{Hf_{y}} \right)^{2}\alpha_{5}}}}.} & (28)\end{matrix}$Ray Direction

With the selected scaling, the unnormalized ray d passing through thepixel becomes

$\begin{matrix}{{d = \left( {x^{\prime},y^{\prime},\frac{Hf_{y}}{2}} \right)},} & (29)\end{matrix}$and its norm∥d∥=√{square root over (x′ ² +y′ ² +q)},  (30)where

$\begin{matrix}{q = \left( \frac{Hf_{y}}{2} \right)^{2}} & (31)\end{matrix}$Range Coefficients

In one embodiment, range coefficients assume the form

$\begin{matrix}{{T_{1} = {p_{1}{d}}}{T_{2} = {p_{2}{d}}}{T_{3} = {{p_{3}x^{\prime}} + {p_{4}y^{\prime}} + p_{5}}}{T_{4} = {{p_{6}x^{\prime}} + {p_{7}y^{\prime}} + p_{8}}}} & (32)\end{matrix}$ $\begin{matrix}{{where}{p_{1} = {\left( P_{p} \right)_{24}s_{1}}}{p_{2} = {{- s_{1}}{s_{2}\left( {\left( P_{p} \right)_{14} + \left( P_{p} \right)_{24}} \right)}}}{p_{3} = {- \left( P_{p} \right)_{21}}}{p_{4} = {- \left( P_{p} \right)_{22}}}{p_{5} = {{- \left( P_{p} \right)_{23}}\frac{H}{2}f_{y}}}{p_{6} = {s_{2}\left( {\left( P_{p} \right)_{11} + \left( P_{p} \right)_{21}} \right)}}{p_{7} = {s_{2}\left( {\left( P_{p} \right)_{12} + \left( P_{p} \right)_{22}} \right)}}{p_{8} = {{s_{2}\left( {\left( P_{p} \right)_{13} + \left( P_{p} \right)_{23}} \right)}\frac{H}{2}f_{y}}}} & (33)\end{matrix}$are precomputed from the system parameters. Depth reconstruction isobtained withT ₁ ′=p ₁′T ₂ =p ₂′  (34)wherep ₁ ′=p ₁′√{square root over (q)}p ₂ ′=p ₂′√{square root over (q)}.  (35)Texture Map Coefficients

Texture map coefficients are precomputed as

$\begin{matrix}{{\sigma_{1} = {x^{\prime} + {h_{0}y^{\prime}} + h_{i}}}{\sigma_{2} = {{h_{2}x^{\prime}} + {h_{3}y^{\prime}} + h_{4}}}{\sigma_{3} = {{h_{5}x^{\prime}} + {h_{6}y^{\prime}} + h_{7}}}{\sigma_{4} = {h_{8}{d}}}{\sigma_{5} = {h_{9}{d}}}{{\sigma_{6} = {h_{10}{d}}},}} & (36)\end{matrix}$ $\begin{matrix}{{where}{h_{0} = \frac{\left( P_{t} \right)_{12} + \left( P_{t} \right)_{32}}{\left( P_{t} \right)_{11} + \left( P_{t} \right)_{31}}}{h_{1} = {\frac{\left( P_{t} \right)_{13} + \left( P_{t} \right)_{33}}{\left( P_{t} \right)_{11} + \left( P_{t} \right)_{31}}\frac{H}{2}f_{y}}}{h_{2} = \frac{\left( P_{t} \right)_{21} + \left( P_{t} \right)_{31}}{\left( P_{t} \right)_{11} + \left( P_{t} \right)_{31}}}{h_{3} = \frac{\left( P_{t} \right)_{22} + \left( P_{t} \right)_{32}}{\left( P_{t} \right)_{11} + \left( P_{t} \right)_{31}}}{h_{4} = {\frac{\left( P_{t} \right)_{23} + \left( P_{t} \right)_{33}}{\left( P_{t} \right)_{11} + \left( P_{t} \right)_{31}}\frac{H}{2}f_{y}}}{h_{5} = {\frac{\left( P_{t} \right)_{31}}{\left( P_{t} \right)_{11} + \left( P_{t} \right)_{31}}\frac{2}{s_{uv}}}}{h_{6} = {\frac{\left( P_{t} \right)_{32}}{\left( P_{t} \right)_{11} + \left( P_{t} \right)_{31}}\frac{2}{s_{uv}}}}{h_{7} = {\frac{\left( P_{t} \right)_{33}}{\left( P_{t} \right)_{11} + \left( P_{t} \right)_{31}}\frac{H}{s_{uv}}f_{y}}}{h_{8} = {\frac{\left( P_{t} \right)_{14} + \left( P_{t} \right)_{34}}{\left( P_{t} \right)_{11} + \left( P_{t} \right)_{31}}\frac{H}{2}s_{1}}}{h_{9} = {\frac{\left( P_{t} \right)_{24} + \left( P_{t} \right)_{34}}{\left( P_{t} \right)_{11} + \left( P_{t} \right)_{31}}s_{1}}}{h_{10} = {2\frac{\left( P_{t} \right)_{34}}{\left( P_{t} \right)_{11} + \left( P_{t} \right)_{31}}{\frac{s_{1}}{s_{uv}}.}}}} & (37)\end{matrix}$When texture map is reconstructed from z rather than from r,σ₄ ′=h ₈′σ₅ ′=h ₉′σ₆ ′=h ₁₀′  (38)are used withh ₈ ′=h ₈√{square root over (q)}h ₉ ′=h ₉√{square root over (q)}h ₁₀ ′=h ₁₀√{square root over (q)}  (39)

FIG. 4 illustrates an example of coefficient precomputation. FIG. 5illustrates an example of range and texture reconstruction andcoefficient precomputation.

FIG. 6 illustrates, for one embodiment, an example system 600 having oneor more processor(s) 604, system control module 608 coupled to at leastone of the processor(s) 604, system memory 612 coupled to system controlmodule 608, non-volatile memory (NVM)/storage 614 coupled to systemcontrol module 608, and one or more communications interface(s) 620coupled to system control module 608. In some embodiments, the system600 may include capture device 100 and provide logic/module thatperforms functions aimed at compensating for temperature in the depthdetermination and correcting distortions in a reconstructed object imagedescribed herein.

In some embodiments, the system 600 may include one or morecomputer-readable media (e.g., system memory or NVM/storage 614) havinginstructions and one or more processors (e.g., processor(s) 604) coupledwith the one or more computer-readable media and configured to executethe instructions to implement a module to perform image distortioncorrection calculation actions described herein.

System control module 608 for one embodiment may include any suitableinterface controllers to provide for any suitable interface to at leastone of the processor(s) 604 and/or to any suitable device or componentin communication with system control module 608.

System control module 608 may include memory controller module 610 toprovide an interface to system memory 612. The memory controller module610 may be a hardware module, a software module, and/or a firmwaremodule. System memory 612 may be used to load and store data and/orinstructions, for example, for system 600. System memory 612 for oneembodiment may include any suitable volatile memory, such as suitableDRAM, for example. System control module 608 for one embodiment mayinclude one or more input/output (I/O) controller(s) to provide aninterface to NVM/storage 614 and communications interface(s) 620.

The NVM/storage 614 may be used to store data and/or instructions, forexample. NVM/storage 614 may include any suitable non-volatile memory,such as flash memory, for example, and/or may include any suitablenon-volatile storage device(s), such as one or more hard disk drive(s)(HDD(s)), one or more compact disc (CD) drive(s), and/or one or moredigital versatile disc (DVD) drive(s), for example. The NVM/storage 614may include a storage resource physically part of a device on which thesystem 600 is installed or it may be accessible by, but not necessarilya part of, the device. For example, the NVM/storage 614 may be accessedover a network via the communications interface(s) 620.

Communications interface(s) 620 may provide an interface for system 600to communicate over one or more network(s) and/or with any othersuitable device. The system 600 may wirelessly communicate with the oneor more components of the wireless network in accordance with any of oneor more wireless network standards and/or protocols.

For one embodiment, at least one of the processor(s) 604 may be packagedtogether with logic for one or more controller(s) of system controlmodule 608, e.g., memory controller module 610. For one embodiment, atleast one of the processor(s) 604 may be packaged together with logicfor one or more controllers of system control module 608 to form aSystem in Package (SiP). For one embodiment, at least one of theprocessor(s) 604 may be integrated on the same die with logic for one ormore controller(s) of system control module 608. For one embodiment, atleast one of the processor(s) 604 may be integrated on the same die withlogic for one or more controller(s) of system control module 608 to forma System on Chip (SoC).

In various embodiments, the system 600 may have more or less components,and/or different architectures. For example, in some embodiments, thesystem 600 may include one or more of a camera, a keyboard, liquidcrystal display (LCD) screen (including touch screen displays),non-volatile memory port, multiple antennas, graphics chip,application-specific integrated circuit (ASIC), and speakers.

In various implementations, the system 600 may be, but is not limitedto, a mobile computing device (e.g., a laptop computing device, ahandheld computing device, a tablet, a netbook, etc.), a laptop, anetbook, a notebook, an ultrabook, a smartphone, a tablet, a personaldigital assistant (PDA), an ultra mobile PC, a mobile phone, a desktopcomputer, a server, a printer, a scanner, a monitor, a set-top box, anentertainment control unit, a digital camera, a portable music player,or a digital video recorder. In further implementations, the system 600may be any other electronic device.

FIG. 7 illustrates an embodiment of a computing environment 700 capableof supporting the operations discussed above. The modules describedbefore can use the depth information (e.g., values) and other datadescribed above to perform these functions. The modules and systems canbe implemented in a variety of different hardware architectures and formfactors.

Command Execution Module 701 includes a central processing unit to cacheand execute commands and to distribute tasks among the other modules andsystems shown. It may include an instruction stack, a cache memory tostore intermediate and final results, and mass memory to storeapplications and operating systems. Command Execution Module 701 mayalso serve as a central coordination and task allocation unit for thesystem.

Screen Rendering Module 721 draws objects on the one or more multiplescreens for the user to see. It can be adapted to receive the data fromVirtual Object Behavior Module 704, described below, and to render thevirtual object and any other objects and forces on the appropriatescreen or screens. Thus, the data from Virtual Object Behavior Module704 would determine the position and dynamics of the virtual object andassociated gestures, forces and objects, for example, and ScreenRendering Module 721 would depict the virtual object and associatedobjects and environment on a screen, accordingly. Screen RenderingModule 721 could further be adapted to receive data from Adjacent ScreenPerspective Module 707, described below, to either depict a targetlanding area for the virtual object if the virtual object could be movedto the display of the device with which Adjacent Screen PerspectiveModule 707 is associated. Thus, for example, if the virtual object isbeing moved from a main screen to an auxiliary screen, Adjacent ScreenPerspective Module 707 could send data to the Screen Rendering Module721 to suggest, for example in shadow form, one or more target landingareas for the virtual object on that track to a user's hand movements oreye movements.

Object and Gesture Recognition System 722 may be adapted to recognizeand track hand and harm gestures of a user. Such a module may be used torecognize hands, fingers, finger gestures, hand movements and a locationof hands relative to displays. For example, Object and GestureRecognition System 722 could for example determine that a user made abody part gesture to drop or throw a virtual object onto one or theother of the multiple screens, or that the user made a body part gestureto move the virtual object to a bezel of one or the other of themultiple screens. Object and Gesture Recognition System 722 may becoupled to a camera or camera array, a microphone or microphone array, atouch screen or touch surface, or a pointing device, or some combinationof these items, to detect gestures and commands from the user.

The touch screen or touch surface of Object and Gesture RecognitionSystem 722 may include a touch screen sensor. Data from the sensor maybe fed to hardware, software, firmware or a combination of the same tomap the touch gesture of a user's hand on the screen or surface to acorresponding dynamic behavior of a virtual object. The sensor date maybe used to momentum and inertia factors to allow a variety of momentumbehavior for a virtual object based on input from the user's hand, suchas a swipe rate of a user's finger relative to the screen. Pinchinggestures may be interpreted as a command to lift a virtual object fromthe display screen, or to begin generating a virtual binding associatedwith the virtual object or to zoom in or out on a display. Similarcommands may be generated by Object and Gesture Recognition System 722,using one or more cameras, without the benefit of a touch surface.

Direction of Attention Module 723 may be equipped with cameras or othersensors to track the position or orientation of a user's face or hands.When a gesture or voice command is issued, the system can determine theappropriate screen for the gesture. In one example, a camera is mountednear each display to detect whether the user is facing that display. Ifso, then the direction of attention module information is provided toObject and Gesture Recognition Module 722 to ensure that the gestures orcommands are associated with the appropriate library for the activedisplay. Similarly, if the user is looking away from all of the screens,then commands can be ignored.

Device Proximity Detection Module 725 can use proximity sensors,compasses, GPS (global positioning system) receivers, personal areanetwork radios, and other types of sensors, together with triangulationand other techniques to determine the proximity of other devices. Once anearby device is detected, it can be registered to the system and itstype can be determined as an input device or a display device or both.For an input device, received data may then be applied to Object Gestureand Recognition System 722. For a display device, it may be consideredby Adjacent Screen Perspective Module 707.

Virtual Object Behavior Module 704 is adapted to receive input fromObject Velocity and Direction Module 703, and to apply such input to avirtual object being shown in the display. Thus, for example, Object andGesture Recognition System 722 would interpret a user gesture and bymapping the captured movements of a user's hand to recognized movements,Virtual Object Tracker Module 706 would associate the virtual object'sposition and movements to the movements as recognized by Object andGesture Recognition System 722, Object and Velocity and Direction Module703 would capture the dynamics of the virtual object's movements, andVirtual Object Behavior Module 704 would receive the input from Objectand Velocity and Direction Module 703 to generate data that would directthe movements of the virtual object to correspond to the input fromObject and Velocity and Direction Module 703.

Virtual Object Tracker Module 706 on the other hand may be adapted totrack where a virtual object should be located in three-dimensionalspace in a vicinity of a display, and which body part of the user isholding the virtual object, based on input from Object Gesture andRecognition System 722. Virtual Object Tracker Module 706 may forexample track a virtual object as it moves across and between screensand track which body part of the user is holding that virtual object.Tracking the body part that is holding the virtual object allows acontinuous awareness of the body part's air movements, and thus aneventual awareness as to whether the virtual object has been releasedonto one or more screens.

Gesture to View and Screen Synchronization Module 708, receives theselection of the view and screen or both from Direction of AttentionModule 723 and, in some cases, voice commands to determine which view isthe active view and which screen is the active screen. It then causesthe relevant gesture library to be loaded for Object and GestureRecognition System 722. Various views of an application on one or morescreens can be associated with alternative gesture libraries or a set ofgesture templates for a given view.

Adjacent Screen Perspective Module 707, which may include or be coupledto Device Proximity Detection Module 725, may be adapted to determine anangle and position of one display relative to another display. Aprojected display includes, for example, an image projected onto a wallor screen. The ability to detect a proximity of a nearby screen and acorresponding angle or orientation of a display projected therefrom mayfor example be accomplished with either an infrared emitter andreceiver, or electromagnetic or photo-detection sensing capability. Fortechnologies that allow projected displays with touch input, theincoming video can be analyzed to determine the position of a projecteddisplay and to correct for the distortion caused by displaying at anangle. An accelerometer, magnetometer, compass, or camera can be used todetermine the angle at which a device is being held while infraredemitters and cameras could allow the orientation of the screen device tobe determined in relation to the sensors on an adjacent device. AdjacentScreen Perspective Module 707 may, in this way, determine coordinates ofan adjacent screen relative to its own screen coordinates. Thus, theAdjacent Screen Perspective Module may determine which devices are inproximity to each other, and further potential targets for moving one ormore virtual object's across screens. Adjacent Screen Perspective Module707 may further allow the position of the screens to be correlated to amodel of three-dimensional space representing all of the existingobjects and virtual objects.

Object and Velocity and Direction Module 703 may be adapted to estimatethe dynamics of a virtual object being moved, such as its trajectory,velocity (whether linear or angular), momentum (whether linear orangular), etc. by receiving input from Virtual Object Tracker Module706. The Object and Velocity and Direction Module 703 may further beadapted to estimate dynamics of any physics forces, by for exampleestimating the acceleration, deflection, degree of stretching of avirtual binding, etc. and the dynamic behavior of a virtual object oncereleased by a user's body part. Object and Velocity and Direction Module703 may also use image motion, size and angle changes to estimate thevelocity of objects, such as the velocity of hands and fingers

Momentum and Inertia Module 702 can use image motion, image size, andangle changes of objects in the image plane or in a three-dimensionalspace to estimate the velocity and direction of objects in the space oron a display. Momentum and Inertia Module 702 is coupled to Object andGesture Recognition System 722 to estimate the velocity of gesturesperformed by hands, fingers, and other body parts and then to applythose estimates to determine momentum and velocities to virtual objectsthat are to be affected by the gesture.

3D Image Interaction and Effects Module 705 tracks user interaction with3D images that appear to extend out of one or more screens. Theinfluence of objects in the z-axis (towards and away from the plane ofthe screen) can be calculated together with the relative influence ofthese objects upon each other. For example, an object thrown by a usergesture can be influenced by 3D objects in the foreground before thevirtual object arrives at the plane of the screen. These objects maychange the direction or velocity of the projectile or destroy itentirely. The object can be rendered by the 3D Image Interaction andEffects Module 705 in the foreground on one or more of the displays.

In a first example embodiment, an apparatus comprises a first cameraconfigured to capture a sequence of images of the object, a seconddevice comprising a projector or a camera, a processing unit to receivethe sequence of images and determine depth information in response toparameters of the camera and the second device, one or more temperaturesensors; and a thermal correction unit responsive to temperatureinformation from the one or more temperature sensors to adjust one ormore of the calibration parameters of the first camera and the seconddevice.

In another example embodiment, the subject matter of the first exampleembodiment can optionally include that the second device comprises aprojector, and further wherein the calibration parameters characterizean optical system formed by the projector and the first camera andcomprise intrinsic parameters of the first camera and the projector andextrinsic parameters of first camera. In another example embodiment, thesubject matter of the last example embodiment can optionally includethat the intrinsic parameters comprise one or more of a focal distanceof the first camera, a principal point of projection of the projector,and a distortion model and wherein the extrinsic parameters comprise arelative location and orientation of the projector with respect to thefirst camera.

In another example embodiment, the subject matter of the first exampleembodiment can optionally include that the processing unit is configuredto perform a triangulation process using the calibration parameters. Inanother example embodiment, the subject matter of the last exampleembodiment can optionally include that the calibration parameters areinitialized by one or more calibration procedures and further whereinthe thermal correction unit is operable to adjust one or more of thecalibration parameters based on outputs from a thermal correction modelthat is responsive to the temperature information from the one or moretemperature sensors.

In another example embodiment, the subject matter of the first exampleembodiment can optionally include that the first camera comprises adigital infrared camera.

In another example embodiment, the subject matter of the first exampleembodiment can optionally include that the second device comprises aprojector configured to project a sequence of light patterns on anobject.

In another example embodiment, the subject matter of the first exampleembodiment can optionally include that the projector is configured toproject one-dimensional coded light patterns on the object.

In another example embodiment, the subject matter of the first exampleembodiment can optionally include a second camera configured to capturea second image of the object illuminated with the projected lightpatterns and configured to capture a texture image of the object in thesecond image. In another example embodiment, the subject matter of thefirst example embodiment can optionally include that the second devicecomprises an infrared (IR) projector, the first camera comprises an IRcamera, and the second camera comprises a red, green, and blue (RGB)camera.

In a second example embodiment, a method comprises capturing, using afirst camera, a sequence of images of the object; and determining depthinformation using calibration parameters of the camera and a seconddevice, including adjusting, based on temperature information from oneor more temperature sensors, one or more of the calibration parametersof the first camera and the second device, where the second devicecomprises a projector or a camera.

In another example embodiment, the subject matter of the second exampleembodiment can optionally include that the second device comprises aprojector that projects the light patterns and further wherein thecalibration parameters characterize an optical system formed by theprojector and the first camera and comprise intrinsic parameters of thefirst camera and the projector and extrinsic parameters of first camera.

In another example embodiment, the subject matter of the second exampleembodiment can optionally include that the intrinsic parameters compriseone or more of a focal distance of the first camera, a principal pointof projection of the projector, and a distortion model and wherein theextrinsic parameters comprise a relative location and orientation of theprojector with respect to the first camera.

In another example embodiment, the subject matter of the second exampleembodiment can optionally include that determining depth informationcomprises performing a triangulation process using the calibrationparameters. In another example embodiment, the subject matter of thelast example embodiment can optionally include projecting, using aprojector, a sequence of light patterns on an object, and wherein thecalibration parameters are initialized by one or more calibrationprocedures and further wherein adjusting the one or more of thecalibration parameters of the projector and the camera is based onoutputs from a thermal correction model that is responsive to thetemperature information from the one or more temperature sensors.

In a third example embodiment, an article of manufacture has one or morenon-transitory computer readable storage media storing instructionswhich when executed by a system to perform a method comprises:projecting a sequence of light patterns on an object; capturing asequence of images of the object illuminated with the projected lightpatterns; determining depth information in response to parameters of theprojector and camera; and adjusting, based on temperature informationfrom one or more temperature sensors, one or more of the calibrationparameters of the projector and the camera.

In another example embodiment, the subject matter of the third exampleembodiment can optionally include that the calibration parameterscharacterize an optical system formed by the projector and the firstcamera and comprise intrinsic parameters of the first camera and theprojector and extrinsic parameters of first camera.

In another example embodiment, the subject matter of the last exampleembodiment can optionally include that the intrinsic parameters compriseone or more of a focal distance of the first camera, a principal pointof projection of the projector, and a distortion model and wherein theextrinsic parameters comprise a relative location and orientation of theprojector with respect to the first camera.

In another example embodiment, the subject matter of the third exampleembodiment can optionally include that determining depth informationcomprises performing a triangulation process using the calibrationparameters.

In another example embodiment, the subject matter of the third exampleembodiment can optionally include that the calibration parameters areinitialized by one or more calibration procedures and further whereinadjusting the one or more of the calibration parameters of the projectorand the camera is based on outputs from a thermal correction model thatis responsive to the temperature information from the one or moretemperature sensors.

Some portions of the detailed descriptions above are presented in termsof algorithms and symbolic representations of operations on data bitswithin a computer memory. These algorithmic descriptions andrepresentations are the means used by those skilled in the dataprocessing arts to most effectively convey the substance of their workto others skilled in the art. An algorithm is here, and generally,conceived to be a self-consistent sequence of steps leading to a desiredresult. The steps are those requiring physical manipulations of physicalquantities. Usually, though not necessarily, these quantities take theform of electrical or magnetic signals capable of being stored,transferred, combined, compared, and otherwise manipulated. It hasproven convenient at times, principally for reasons of common usage, torefer to these signals as bits, values, elements, symbols, characters,terms, numbers, or the like.

It should be borne in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise as apparent from the following discussion,it is appreciated that throughout the description, discussions utilizingterms such as “processing” or “computing” or “calculating” or“determining” or “displaying” or the like, refer to the action andprocesses of a computer system, or similar electronic computing device,that manipulates and transforms data represented as physical(electronic) quantities within the computer system's registers andmemories into other data similarly represented as physical quantitieswithin the computer system memories or registers or other suchinformation storage, transmission or display devices.

The present invention also relates to apparatus for performing theoperations herein. This apparatus may be specially constructed for therequired purposes, or it may comprise a general purpose computerselectively activated or reconfigured by a computer program stored inthe computer. Such a computer program may be stored in a computerreadable storage medium, such as, but is not limited to, any type ofdisk including floppy disks, optical disks, CD-ROMs, andmagnetic-optical disks, read-only memories (ROMs), random accessmemories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, or any typeof media suitable for storing electronic instructions, and each coupledto a computer system bus.

The algorithms and displays presented herein are not inherently relatedto any particular computer or other apparatus. Various general purposesystems may be used with programs in accordance with the teachingsherein, or it may prove convenient to construct more specializedapparatus to perform the required method steps. The required structurefor a variety of these systems will appear from the description below.In addition, the present invention is not described with reference toany particular programming language. It will be appreciated that avariety of programming languages may be used to implement the teachingsof the invention as described herein.

A machine-readable medium includes any mechanism for storing ortransmitting information in a form readable by a machine (e.g., acomputer). For example, a machine-readable medium includes read onlymemory (“ROM”); random access memory (“RAM”); magnetic disk storagemedia; optical storage media; flash memory devices; etc.

Whereas many alterations and modifications of the present invention willno doubt become apparent to a person of ordinary skill in the art afterhaving read the foregoing description, it is to be understood that anyparticular embodiment shown and described by way of illustration is inno way intended to be considered limiting. Therefore, references todetails of various embodiments are not intended to limit the scope ofthe claims which in themselves recite only those features regarded asessential to the invention.

We claim:
 1. An apparatus comprising: means for modifying a calibrationparameter of a capture device based on a compensation value, thecompensation value corresponding to a temperature of the capture device;and means for generating a depth value based on a pixel location of anobject in an image captured by the capture device, a location of aprojected plane in the image, and the calibration parameter, the depthvalue corresponding to a distance to the object in the image.
 2. Theapparatus of claim 1, wherein the means for generating the depth valueis to generate the depth value based on an intersection of the projectedplane and a ray projected by the capture device.
 3. The apparatus ofclaim 1, further including means for generating the projected plane inthe image by projecting at least one of a one-dimensional coded lightpattern or a two-dimensional coded light pattern.
 4. The apparatus ofclaim 1, wherein the means for generating the depth value is to generatethe depth value based on at least one of an intrinsic parameter or anextrinsic parameter of the capture device.
 5. The apparatus of claim 4,wherein the intrinsic parameter is a focal distance of the capturedevice, a principal point of projection of a projector that is togenerate the projected plane, a distortion model, or a characteristic ofan optical system formed by the projector and the capture device.
 6. Theapparatus of claim 4, wherein the extrinsic parameter is at least one ofa relative location of a projector or an orientation of the projectorwith respect to a camera of the capture device, the camera to capturethe image, the projector to generate the projected plane.
 7. Theapparatus of claim 1, wherein the means for modifying the calibrationparameter is to modify the calibration parameter for ones of framescorresponding to a frame rate of the capture device.
 8. The apparatus ofclaim 1, wherein the pixel location is a first pixel location, the imageis a first image, and the capture device includes a first camera tocapture the first image, the means for generating the depth value togenerate the depth value based on (1) the first pixel location, (2) asecond pixel location of the object in a second image captured by asecond camera of the capture device, and (3) the calibration parameter,the depth value corresponding to the distance to the object based on thefirst image and the second image.
 9. A capture device comprising:memory; instructions; and processor circuitry to execute theinstructions to: modify a calibration parameter of the capture devicebased on a compensation value, the compensation value corresponding to atemperature of the capture device; and generate a depth value based on apixel location of an object in an image captured by the capture device,a location of a projected plane in the image, and the calibrationparameter, the depth value corresponding to a distance to the object inthe image.
 10. The capture device of claim 9, wherein the processorcircuitry is to execute the instructions to generate the depth valuebased on an intersection of the projected plane and a ray projected bythe capture device.
 11. The capture device of claim 9, wherein theprocessor circuitry is to execute the instructions to generate theprojected plane in the image by projecting at least one of aone-dimensional coded light pattern or a two-dimensional coded lightpattern.
 12. The capture device of claim 9, wherein the processorcircuitry is to execute the instructions to generate the depth valuebased on at least one of an intrinsic parameter or an extrinsicparameter of the capture device.
 13. The capture device of claim 12,wherein the intrinsic parameter is a focal distance of the capturedevice, a principal point of projection of a projector that is togenerate the projected plane, a distortion model, or a characteristic ofan optical system formed by the projector and the capture device. 14.The capture device of claim 12, wherein the extrinsic parameter is atleast one of a relative location of a projector or an orientation of theprojector with respect to a camera of the capture device, the camera tocapture the image, the projector to generate the projected plane. 15.The capture device of claim 9, wherein the processor circuitry is toexecute the instructions to modify the calibration parameter for ones offrames corresponding to a frame rate of the capture device.
 16. Thecapture device of claim 9, wherein the pixel location is a first pixellocation, the image is a first image, and the capture device includes afirst camera to capture the first image, the processor circuitry toexecute the instructions to generate the depth value based on (1) thefirst pixel location, (2) a second pixel location of the object in asecond image captured by a second camera of the capture device, and (3)the calibration parameter, the depth value corresponding to the distanceto the object based on the first image and the second image.
 17. Amethod comprising: modifying, by executing an instruction with aprocessor, a calibration parameter of a capture device based on acompensation value, the compensation value corresponding to atemperature of the capture device; and generating, by executing aninstruction with the processor, a depth value based on a pixel locationof an object in an image captured by the capture device, a location of aprojected plane in the image, and the calibration parameter, the depthvalue corresponding to a distance to the object in the image.
 18. Themethod of claim 17, wherein the generating of the depth value is basedon an intersection of the projected plane and a ray projected by thecapture device.
 19. The method of claim 17, wherein the generating ofthe depth value is based on at least one of an intrinsic parameter or anextrinsic parameter of the capture device, the intrinsic parameter is afocal distance of the capture device, a principal point of projection ofa projector that is to generate the projected plane, a distortion model,or a characteristic of an optical system formed by the projector and thecapture device, the extrinsic parameter is at least one of a relativelocation of the projector or an orientation of the projector withrespect to a camera of the capture device, wherein the camera is tocapture the image, and the projector is to generate the projected plane.20. The method of claim 17, wherein the pixel location is a first pixellocation, the image is a first image, and the capture device includes afirst camera to capture the first image, the generating of the depthvalue based on (1) the first pixel location, (2) a second pixel locationof the object in a second image captured by a second camera of thecapture device, and (3) the calibration parameter, the depth valuecorresponding to the distance to the object based on the first image andthe second image.